8 research outputs found
"Zero-Shot" Super-Resolution using Deep Internal Learning
Deep Learning has led to a dramatic leap in Super-Resolution (SR) performance
in the past few years. However, being supervised, these SR methods are
restricted to specific training data, where the acquisition of the
low-resolution (LR) images from their high-resolution (HR) counterparts is
predetermined (e.g., bicubic downscaling), without any distracting artifacts
(e.g., sensor noise, image compression, non-ideal PSF, etc). Real LR images,
however, rarely obey these restrictions, resulting in poor SR results by SotA
(State of the Art) methods. In this paper we introduce "Zero-Shot" SR, which
exploits the power of Deep Learning, but does not rely on prior training. We
exploit the internal recurrence of information inside a single image, and train
a small image-specific CNN at test time, on examples extracted solely from the
input image itself. As such, it can adapt itself to different settings per
image. This allows to perform SR of real old photos, noisy images, biological
data, and other images where the acquisition process is unknown or non-ideal.
On such images, our method outperforms SotA CNN-based SR methods, as well as
previous unsupervised SR methods. To the best of our knowledge, this is the
first unsupervised CNN-based SR method
Idempotent Generative Network
We propose a new approach for generative modeling based on training a neural
network to be idempotent. An idempotent operator is one that can be applied
sequentially without changing the result beyond the initial application, namely
. The proposed model is trained to map a source distribution
(e.g, Gaussian noise) to a target distribution (e.g. realistic images) using
the following objectives: (1) Instances from the target distribution should map
to themselves, namely . We define the target manifold as the set of all
instances that maps to themselves. (2) Instances that form the source
distribution should map onto the defined target manifold. This is achieved by
optimizing the idempotence term, which encourages the range of
to be on the target manifold. Under ideal assumptions such a process
provably converges to the target distribution. This strategy results in a model
capable of generating an output in one step, maintaining a consistent latent
space, while also allowing sequential applications for refinement.
Additionally, we find that by processing inputs from both target and source
distributions, the model adeptly projects corrupted or modified data back to
the target manifold. This work is a first step towards a ``global projector''
that enables projecting any input into a target data distribution
InGAN: Capturing and Retargeting the “DNA” of a Natural Image
Generative Adversarial Networks (GANs) typically learn a distribution of images in a large image dataset, and are then able to generate new images from this distribution. However, each natural image has its own internal statistics, captured by its unique distribution of patches. In this paper we propose an ''Internal GAN'' (InGAN) - an image-specific GAN - which trains on a single input image and learns its internal distribution of patches. It is then able to synthesize a plethora of new natural images of significantly different sizes, shapes and aspect-ratios - all with the same internal patch-distribution (same ''DNA'') as the input image. In particular, despite large changes in global size/shape of the image, all elements inside the image maintain their local size/shape. InGAN is fully unsupervised, requiring no additional data other than the input image itself. Once trained on the input image, it can remap the input to any size or shape in a single feedforward pass, while preserving the same internal patch distribution. InGAN provides a unified framework for a variety of tasks, bridging the gap between textures and natural images